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1. Introduction 


Debate about the performance of healthcare systems has been amplified by the current 
Covid19 pandemic. The impact of this crisis has served to highlight the fragility of many such 
systems and the key need for policymakers and health service managers across the world to 
evaluate their performance. Arguably the strategic development of any healthcare system 
should aim to reduce health inequalities, and therefore, as a minimum, it is necessary to 
monitor its performance in addressing inequalities in both health and its social determinants. 
The situation in Italy is a case in point, where there are demands for better quality of care, 
higher productivity, better responsiveness, more efficiency and better sustainability. All of 
these are expressions of the same question, viz. how to improve the performance of health 
services and health workers? However, measuring healthcare performance presents 
difficulties because of its multidimensional nature, which can easily lead to conceptual and 
methodological confusion. As a consequence, there is a scarcity of models which fully 
analyse performance at healthcare system level. Unsurprisingly, virtually all current 
performance frameworks include quality of care as a key element, with effectiveness, 
productivity and efficiency also being recurrent themes. Examples include the World Health 
Organisation’s (WHO) World Health Report 2000, the Organisation for Economic Co- 
operation and Development’s (OECD) framework (2004), and the Nuti’s framework (2008). 
In contrast, social outcomes of healthcare and equity are missing or little developed in most 
frameworks, with Australian and Canadian national frameworks being notable exceptions. 
Given this situation, Sicotte et al. (1999) developed the comprehensive Evaluation Globale et 
Intégrée de la Performance des Systèmes de Santé (EGIPPS) framework for the assessment of 
the performance of Health Care Organizations (HCOs). Therefore, the main aims of this paper 
are: 

e To describe the key features of the EGIPPS framework; 

e To present the authors’ version of the EGIPSS framework; and 

e To illustrate how it can be used, with reference to an “Area Vasta” of Italy’s 
Marche region, to the Republic of San Marino and to other territories. 


2. Key features of the EGIPSS model 


In the healthcare sector, one framework stands out: the EGIPSS framework developed by 
Sicotte et al. (cit.), which is a comprehensive approach to the assessment of performance of 
HCOs. It includes goal achievement, service production and adaptation to the 
environment as core dimensions of performance, and usefully adds a focus on values and 
culture. EGIPSS is geared towards North American settings and has been mainly used in 
OECD countries. For example, it acts as the basis of WHO-Europe’s framework for assessing 
hospitals, to assess accreditation schemes, to analyse how actors and stakeholders of an HCO 
define performance and to explore how HCOs learn. 

In this paper, the authors present a practical, simplified version of the EGIPSS framework. 
Keeping the key strengths of this framework, some elements were redefined based on 
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concepts of integrated healthcare systems and public service. Inspiration was found in 
Parsons’ social system action theory to develop an integrative framework of performance, 
with the performance of a HCO considered to be multi-dimensional. More specifically, it is 
the result of the interaction between four organisational functions (see Figure 1). 
Consequently, the success of an organisation depends not only on how each of these functions 
is organised, but also on how they are aligned with each other. Performance is therefore 
understood as something more comprehensive than merely efficiently producing desired 
outputs. Furthermore, it incorporates the managerial approach of the New Public Management 
(NPM). The framework also describes six equilibriums or alignments between these four 
functions, which can be best understood as tensions that may arise between the functions as a 
result of a change in one of them (Figure 1). 

The tactical alignment links the Goal Achievement and Service Production function. 
This deals first with the appropriateness of the service provision in relation to the goals: “To 
what extent do the service production processes contribute to attaining the goals? Are they 
effectively producing the output needed to reach the goals? ”. 

The allocative alignment links the Interaction with the environment and the Service 
Production function. It first deals with resource acquisition. Questions that can be used to 
assess this include: “Are the obtained resources adequate to organise the service production 
function? Is the service production function optimal in relation to available resources? ”. 

The strategic alignment examines the link between the Goals that the HCO is pursuing 
and its Environment. Here, questions include whether the organisational goals correspond 
with the needs of the population and other key actors. 

The legitimating alignment is about the congruence of the Goal Attainment function with 
the Culture and Values Maintaining function, and questions how the strategic choice of goals 
influences and shapes the organisational values. 

The operational alignment covers the congruence of the Culture and Values Maintaining 
function with the Service Production modalities, and the impact of the Service Production 
system on the organisational culture and values. 

Finally, the contextual alignment between Culture and Values Maintaining function and 
Adaptation to the environment deals with how the social, political and cultural dimensions of 
the environment influence the organisational culture and its core operational values. 

Figure 1 below sets out the model: 


Fig. 1: EGIPSS model 
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Source: Sicotte et al. (1998) 


3. Materials and methods 


The research made use of various statistical sources (Istituto Nazionale di Statistica 
(ISTAT), Centro Studi Investimenti Sociali, Osservasalute (CENSIS), Istituto Superiore di 
Sanita and an array of survey methods. The indicators used in the performance evaluation 
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model were determined through an in-depth study of the existing literature (Sicotte, cit.) and 
in collaboration with experts from the two locations involved in the study. 

Data relating to the Republic of San Marino was provided by its Health Authority, its 

Istituto per la Sicurezza Sociale (ISS RSM) and by the Office of Statistics of the Republic. 
The data for the “AreaVasta” of the Marche Region were local and regional statistical 
sources, plus some internal information sources. 
Once the list of indicators was identified, a “balise” (according to French terminology) or 
benchmarking (according to English terminology) of excellence was determined for each 
indicator. This represented a norm/guide/method against which results could be compared, 
thereby enabling opinions and judgements to be formed. The EGIPSS model and 
methodology incorporates performance indices that enable comparisons to be made based on 
excellence, which can then be weighted relative to each other within a set of dimensional and 
subdimensional categories. For example, the Adaptation function covers the dimension of 
“Availability of resources’ which has two subdimensions of ‘Healthcare expenditure and 
financing’ and ‘Health workforce’ (see tab.2 below). The weights used were based on the 
original weights provided by the model which were in turn validated by a panel of experts 
representing the various stakeholders of the healthcare systems studied, chosen according to 
their skills. This validation process incorporated the DELPHI method (Fabbris et al 2007). 

One analytical issue involves establishing the relationship between an indicator and 
performance. The approach adopted was to determine a balise of excellence for each 
indicator, that involved values considered to be ‘high performing’. The sense of variation in 
the relationship between an indicator and its associated performance can be positive, negative 
or parabolic. An overall performance achievement index for a subdimension and dimension is 
calculated by applying the assigned weights to the calculated percentage of achievement of 
the balise for each indicator and then aggregating the results. This can be done at each level, 
on the basis that if the weights are expressed as a percentage their sum within a sub- 
dimension, dimension and function must be equal to 100. The process can be repeated for the 
four functions provided by the model. 

Once the percentage of achievement of the balise has been calculated, it is possible to assign a 
qualitative scale of performance. This serves to add precision, with the values used in this 
study shown in table 1: 


Tab.1: Levels of performance 


Level of performance Values 
Very worrying X< 65% 
Worrying 65% < X< 75% 
Good 75% < X < 90% 
Excellent X > 90% 


4. Results 


This section presents a synthesis and summary of the results derived from the application 
of the EGIPSS model in the “Area Vasta” of the Marche region and the Republic of San 
Marino (the latter involves a more reduced version, due to a lack of certain data in local 
information systems). 

Comparisons were made using the indicators relating to the Adaptation function, the 
Service Production function, Goal Attainment, and the Culture and Values Maintaining 
function, which are set out in tabs. 2, 3, 4 and 5, respectively. In addition to the above tables 
the authors sought a helpful diagrammatical presentation that enables the reader to judge the 
relative performance of different healthcare organisations in terms of ‘Strategic equilibrium’. 
Figure 2 below illustrates this for the relationship between Infant Mortality and Healthcare 
Expenditure and Financing for the three healthcare organisations studied. 
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Fig. 2: Strategic equilibrium 
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5. Weaknesses and strengths of the results obtained 


The model provides an overview of the performance, especially for the “Area Vasta” of 
the Marche, through a prism of 107 indicators. These indicators can present warning signs 
regarding accessibility, technical quality, efficiency and fairness of a system. 

Among the inevitable weaknesses that can affect any frameworks of this nature it should 
be pointed out that the evaluation of performance is equivalent to comparing the result of an 
indicator, of a dimension or of a sub-dimension, to a given standard. Where such standards 
exist they have been used. Otherwise, the performance evaluation was set against external 
objectives (such as those determined by Canada, the WHO and OECD) or comparisons with 
the similar results from other countries. The authors’ choice on Canada was justified by the 
fact that this country, by establishing its own empirical standards of excellence, utilised its 
own comparisons with the EU15 countries (Vrijens et al., 2016). This approach, the only 
practicable one available, made it possible to position the areas studied in relation to those 
states that have similar healthcare systems. However, it should be noted that the interpretation 
of relative performance between micro-areas and States appears very delicate due to the 
methodological and contextual differences that can compromise the validity of the 
comparisons. Further constraints were the absence of relevant indicators or the lack of data in 
the information systems of the two areas. 


6. Conclusions 


Performance evaluation is a process that enables the holistic analysis of healthcare 
systems utilising measurable indicators. Its role, therefore, is to improve the quality of the 
decisions being taken by all the staff in the healthcare arena and those services that can impact 
on people’s health. The principles and orientation of the EGIPPS model are useful for 
assessing healthcare systems at any level, whether it be a country, a province or even a local 
community. 
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